Linguistically Driven Multi-Task Pre-Training for Low-Resource Neural Machine Translation

In the present study, we propose novel sequence-to-sequence pre-training objectives for low-resource machine translation (NMT): Japanese-specific sequence to (JASS) language pairs involving Japanese as source or target language, and English-specific (ENSS) English. JASS focuses on masking reordering linguistic units known bunsetsu, whereas ENSS is proposed based phrase structure tasks. Experiments ASPEC Japanese--English & Japanese--Chinese, Wikipedia News English--Korean corpora demonstrate that outperform MASS other existing language-agnostic methods by up +2.9 BLEU points tasks, +7.0 Japanese--Chinese tasks +1.3 Empirical analysis, which relationship between individual parts in ENSS, reveals complementary nature of subtasks ENSS. Adequacy evaluation using LASER, human evaluation, case studies our significantly without injected knowledge they have a larger positive impact adequacy compared fluency. We release codes here: https://github.com/Mao-KU/JASS/tree/master/linguistically-driven-pretraining.